Apparatus for optical see-through head mounted display with mutual occlusion and opaqueness control capability

ABSTRACT

The present invention comprises a compact optical see-through head-mounted display capable of combining, a see-through image path with a virtual image path such that the opaqueness of the see-through image path can be modulated and the virtual image occludes parts of the see-through image and vice versa.

RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No.61/620,574, filed on Apr. 5, 2012 and U.S. Provisional Application No.61/620,581, filed on Apr. 5, 2012, the disclosures of which areincorporated herein by reference in its entirety.

GOVERNMENT LICENSE RIGHTS

This invention was partially made with government support under SBIRcontract No. W91CRB-12-C-0002 awarded by the U.S. ARMY. The governmenthas certain rights in the invention.

FIELD OF THE TINE INVENTION

The present invention relates generally to Head Mounted Displays, andmore particularly, but not exclusively, to optical see-throughhead-mounted displays with opaqueness control and mutual occlusioncapability in which real objects may be occluded by computer-renderedvirtual objects situated in front or vice versa.

BACKGROUND OF THE INVENTION

Over the past decades, Augmented Reality (AR) technology has beenapplied in many application fields, such as medical and militarytraining, engineering design and prototyping, tele-manipulation andtele-presence, and personal entertainment systems. See-throughHead-Mounted Displays (ST-HMD) are one of the enabling technologies ofan augmented reality system for merging virtual views with a physicalscene. There are two types of ST-HMDs: optical and video (J. Rolland andH. Fuchs, “Optical versus video see-through head mounted displays,” InFundamentals of Wearable Computers and Augmented Reality, pp. 113-157,2001.). The major drawbacks of the video see-through approach include:degradation of the image quality of the see-through view; image lag dueto processing of the incoming video stream; potentially loss of thesee-through view due to hardware/software malfunction. In contrast, theoptical see-through HMD (OST-HMD) provides a direct view of the realworld through a beamsplitter and thus has minimal affects to the view ofthe real world. It is highly (preferred in demanding applications wherea user's awareness to the live environment is paramount.

Developing optical see-through HMDs, however, confronts complicatedtechnical challenges. One of the critical issues lies in that thevirtual views in an OST-HMD appear “ghost-like” and are floating in thereal world due to the lack of the occlusion capability. FIG. 1 shows acomparison illustration of the augmented view seen through a typicalOST-HWID (FIG. 1a ) and the augmented view seen through an occlusioncapable OST-HMD (OCOST-HMD) system (FIG. 1b ), In the figure, a virtualcar model is superimposed on a solid platform which represents a realobject. Without proper occlusion management as shown in FIG. 1a , in atypical AR view, the car is mixed with the platform and it is difficultto distinguish the depth relationship of the car and the platform. Onthe contrary, with proper occlusion management as shown in FIG. 1b , thecar blocks a portion of the platform and it can be clearly identifiedthat the car seats on the top of the platform. Adding occlusioncapability to the AR display enables realistically merging virtualobjects into the real environment. Such occlusion-enabled capability maygenerate transformative impacts on AR display technology and is veryappealing for many augmented-reality based applications.

An OCOST-HMD system typically comprises of two key sub-systems. Thefirst is an eyepiece optics that allows a user to see a magnified imagedisplayed on a microdisplay; and the second is a relay optics thatcollects and modulates the light from an external scene in the realworld, which enables the opaqueness and occlusion control on theexternal scene when presenting to the viewers. The key challenges ofcreating truly portable and lightweight OCOST-HMD system lies inaddressing three cornerstone issues: (1) an optical scheme that allowsthe integration of the two subsystems without adding significant weightand volume to the system. (2) a proper optical method that maintains theparity of the coordinate system of the external scene; (3) an opticaldesign method that enables the design of these optical subsystems withan elegant form factor, which has been a persisting dream for HMDdevelopers. Several occlusion-capable optical ST-HMD concepts have beendeveloped (U.S. Pat. No. 7,639,208 B1—Kiyokawa, K., Kurata, Y., andOhno, H., “An Optical See-through Display for Mutual Occlusion with aReal-time Stereo Vision System,” Elsevier Computer & Graphics, SpecialIssue on “Mixed Realities—Beyond. Conventions,” Vol, 25, No. 5, pp.2765-779, 2001. K. Kiyokawa, M, Billinghurst, B. Campbell, E. Woods, “AnOcclusion-Capable Optical See-through Head Mount Display for SupportingCo-located Collaboration,” ISMAR 2003, pp, 133-141). For example,Kiyokawa et. al. developed ELMO series occlusion displays usingconventional lenses, prisms and minors. Not only because of the numberof elements being used, but also more importantly due to therotationally symmetric nature of the optical systems, the existingocclusion-capable OST-HMDs have a helmet-like, bulky form factor. Theyhave been used exclusively in laboratory environments due to the heavyweight and cumbersome design. The cumbersome, helmet-like form factorprevents the acceptance of the technology for many demanding andemerging applications.

SUMMARY OF THE INVENTION

This invention concerns an optical see-through head mounted display(OST-HMD) device with opaqueness control and mutual occlusioncapability, The display system typically comprises of a virtual viewpath for viewing a displayed virtual image and a see-through path forviewing an external scene in the real world. In the present invention,the virtual view path includes a miniature image display unit forsupplying virtual image content and an eyepiece through which a userviews a magnified virtual image. The see-through path comprises of anobjective optics to directly capture the light from the external sceneand firm at least one intermediate image, a spatial light modular (SLM)placed at or near an intermediate image plane in the see-through path tocontrol and modulate the opaqueness of the see-through view, and aneyepiece optics through which the modulated see-through view is seen bythe viewer. In the see-through path, the objective optics and eyepiecetogether act as a relay optics for passing the light from the real worldto viewer's eye. To achieve a compact form factor and reduce theviewpoint offset, the see-through path is folded into two layers throughseveral reflective surfaces, a front layer accepting the incoming lightfrom an external scene and a back layer coupling the light captured bythe front layer into a viewer's eye. The see-through path is merged withthe virtual image path by a beamsplitter so that the same the eyepieceis shared by both paths for viewing displayed virtual content and themodulated see-through image. The microdisplay and the SLM are opticallyconjugate to each other through the beamsplitter, which makes the pixellevel occlusion manipulation possible. In the present invention, theeyepiece, the objective optics, or both may be rotationally symmetriclenses or non-rotationally symmetric freeform optics. In one of itssignificant aspects, the present invention may utilize freeform opticaltechnology in eyepiece optics, objective optics or both to achieve acompact and lightweight OCOST-HMD design.

The reflective surfaces for folding the optical paths may be planarmirrors, spherical, aspherical, or freeform surfaces with optical power.In another significant aspect of the present invention, some of thereflective surfaces may utilize freeform optical technology. Some of thereflective surfaces may also be strategically designed to be an integralpart of the eyepiece or objective optics where the reflective surfacesnot only facilitate the folding of the optical path for achievingcompact display design but also contribute optical power and correctoptical aberrations. In an exemplary configuration, the presentinvention may use a one-reflection or multi-reflection freeform prism asan eyepiece or objective optics where the prism is a single opticalelement comprises of refractive surfaces and one or more than onereflective surfaces for folding the optical path and correctingaberrations.

In another significant aspect of the present invention, the objectiveoptics in the see-through path forms at least one accessibleintermediate image, near which an SLM is placed to provide opaquenesscontrol and see-through modulation. In the present invention, either areflection-type SLM or a transmission-type SLM may be used formodulating the see-through view for occlusion control. A longer backfocal distance for the objective optics is required for areflection-type SLM than a transmission-type SLM. A reflection-type SLMmay have the advantage of higher light efficiency than atransmission-type SLM.

In another significant aspect of the present invention, the see-throughpath may form an odd or even number of intermediate images. In the caseof an odd number of intermediate images, an optical method is providedto invert and/or revert the see-through view in the see-through path.For example, depending on the number of reflections involved in thesee-through path, examples of the possible methods include, but notlimited to, inserting an additional reflection or reflections, utilizinga roof mirror surface, or inserting an erection prism or lens. In thecase of an even number of intermediate images, no image erection elementis needed if there is no parity change in the see-through view. Forinstance, multiple-reflection freeform prism structure (typical morethan 2) may be utilized as eyepiece or objective optics, or both, whichallow folding the see-through optical path inside the objective and/oreyepiece prism multiple times and form intermediate image(s) inside theprisms which eliminates the necessity of using an erection roofreflective surface. The potential advantage of eliminating the erectionprism is that the approach may lead to a more compact design.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing summary and the following detailed description ofexemplary embodiments of the present invention may be further understoodwhen read in conjunction with the appended drawings, in which:

FIG. 1 schematically illustrates AR views seen through an opticalsee-through HMD: without occlusion capability (FIG. 1a ) and withocclusion capability (FIG. 1b ),

FIGS. 2a and 2b schematically illustrate an exemplary optical accordancewith the present invention shown as a monocular optical module.

FIG. 3 schematically illustrates a preferred embodiment in accordancewith the present invention based on freeform optical technology. Theembodiment comprises of a one-reflection eyepiece prism, aone-reflection objective prism, a reflection-type SLM and a roofreflective surface.

FIG. 4 schematically illustrates another preferred embodiment inaccordance with the present invention based on freeform opticaltechnology. The embodiment comprises of a two-reflection eyepiece prism,a four-reflection objective prism, and a reflection-type SLM.

FIG. 5 schematically illustrates another preferred embodiment inaccordance with the present invention based on freeform opticaltechnology. The embodiment comprises of a two-reflection eyepiece prism,a one-reflection objective prism, a transmission-type SLM and a roofreflective surface.

FIG. 6 schematically illustrates another preferred embodiment inaccordance with the present invention based on freeform opticaltechnology. The embodiment comprises of a two-reflection eyepiece prism,a three-reflection objective prism and a transmission-type SLM.

FIG. 7 schematically illustrates another preferred embodiment inaccordance with the present invention based on freeform opticaltechnology. The embodiment comprises of a two-reflection eyepiece prism,a two-reflection objective prism, a reflection-type SLM and a relaylens.

FIG. 8 schematically illustrates an exemplary design of an OCOST-HMDsystem in accordance with the present invention based on an exemplarylayout in FIG. 3.

FIG. 9 illustrates the field map plot of the polychromatic modulationtransfer functions (MTF) of the virtual display path of the design inFIG. 8 at cutoff frequency 401 ps/min (line pairs per millimeter)evaluated using 3 min pupil diameter.

FIG. 10 schematically illustrate an exemplary design of an OCOST-HMDsystem in accordance with the present invention based on an exemplarylayout in FIG. 3 with the eyepiece and objective optics having identicalfreeform structure.

FIG. 11 illustrates the field map plot of the polychromatic modulationtransfer functions (MTF) of the virtual display path of the design inFIG. 10 at cutoff frequency 401 ps/mm (line pairs per millimeter)evaluated using 3 mm pupil diameter.

FIG. 12 depicts a block diagram of an example of an image processingpipeline in accordance with the present invention.

FIG. 13 shows Table 1: Optical surface prescription of surface 1 of theeyepiece prism

FIG. 14 shows Table 2: Optical surface prescription of surface 2 of theeyepiece prism

FIG. 15 shows Table 3: Optical surface prescription of surface 3 ofeyepiece prism

FIG. 16 shows Table 4: Position and orientation parameters of theeyepiece prism

FIG. 17 shows Table 5: Optical surface prescription of surface 4 of theobjective prism

FIG. 18 shows Table 6: Optical surface prescription of surface 5 of theobjective prism

FIG. 19 shows Table 7: Optical surface prescription of surface 6 of theobjective prism

FIG. 20 shows Table 8: Position and orientation parameters of theobjective prism

FIG. 21 shows Table 9: Surface parameters for DOE plates 882 and 884

FIG. 22 shows Table 10: Optical surface prescription of surface 1 of thefreeform prism

FIG. 23 shows Table 11: Optical surface prescription of surface 2 of thefreeform prism

FIG. 24 shows Table 12: Optical surface prescription of surface 3 of thefreeform prism

FIG. 25 shows Table 13: Position and orientation parameters of thefreeform prism as the eyepiece

DETAILED DESCRIPTION OF THE INVENTION

The embodiments according to the present invention will be fullydescribed with respect to the attached drawings. The descriptions areset forth in order to provide an understanding of the invention.However, it will be apparent that the invention can be practiced withoutthese details. Furthermore, the present invention may be implemented invarious forms. However, the embodiments of the present inventiondescribed below shall not be constructed as limited to the embodimentsset forth herein. Rather, these embodiments, drawings and examples areillustrative and are meant to avoid obscuring the invention.

An occlusion capable optical see-through head-mounted display(OCOST-HMD) system typically comprises of a virtual view path forviewing a displayed virtual image and a see-through path for viewing anexternal scene in the real world. Hereafter the virtual image observedthrough the virtual view path is referred to as the virtual view and theexternal scene observed through the see-though path is referred to asthe see-through view. In some embodiments of the present invention, thevirtual view path includes a microdisplay unit for supplying virtualimage content and an eyepiece through which a user views a magnifiedvirtual image. The see-through path comprises of an objective optics tocapture the light from the external scene and form at least oneintermediate image, a spatial light modular (SLM) placed at or near anintermediate image plane in the see-through path to control and modulatethe opaqueness of the see-through view, and an eyepiece through whichthe modulated see-through view is seen by the viewer. In the see-throughpath, the objective optics and eyepiece together act as a relay opticsfor passing the light from the real world to viewer's eye. Theintermediate image in the see-through path is referred to as asee-through image, and an intermediate image modulated by the SLIM isreferred to as a modulated see-through image. An OCOST-HMD produces acombined view of the virtual and see-through views, in which the virtualview occludes portions of the see-through view.

A some embodiment, the present invention comprises a compact opticalsee-through head-mounted display 200, capable of combining a see-throughpath 207 with a virtual view path 205 such that the opaqueness of thesee-through path can be modulated and the virtual view occludes parts ofthe see-through view and vice versa, the display comprising:

-   -   a. a microdisplay 250 for generating an image to be viewed by a        user, the microdisplay having a virtual view path 205 associated        therewith;    -   b. a spatial light modulator 240 for modifying the light from an        external scene in the real world to block portions of the        see-through view that are to be occluded, the spatial light        modulator having a see-through path 207 associated therewith;    -   c. an objective optics 220 configured to receive the incoming        light from the external scene and to focus the light upon the        spatial light modulator 240;    -   d. a beamsplitter 230 configured to merge a virtual image from a        microdisplay 250 and a modulated see-through image of an        external scene passing from a spatial light modulator, producing        a combined image;    -   e. an eyepiece 210 configured to magnify the combined image;    -   f. an exit pupil 202 configured to face the eyepiece, where the        user observes a combined view of the virtual and see-through        views in which the virtual view occludes portions of the        see-through view;    -   g. a plurality of reflective surfaces configured to fold the        virtual view path 205 and see-through paths 207 into two layers.

In some embodiments, at least three reflective surfaces are used to foldthe virtual and see-through paths into two layers. The first reflectivesurface (M1) is located upon the front layer of the display oriented toreflect light from the external scene. The objective optics 220 islocated upon the front layer of the display. The second reflectivesurface (M2) is located upon the front layer of the display oriented toreflect light into the spatial light modulator. The spatial lightmodulator 240 is located at or near an intermediate image plane of thesee-through path 207, in optical communication with the objective optics220 and the eyepiece 210 through the beam splitter 230 along thesee-through path 207. The microdisplay 250 is located at the focal planeof the eyepiece 210, in optical communication with the eyepiece 210through the beamsplitter 230 along the virtual view path 205. The beamsplitter 230 is oriented such that the see-through path 207 is mergedwith virtual view path 205 and the light from both the see-through pathand the virtual view path is directed to the eyepiece 210. The eyepiece210 is located upon the back layer of the display. The third reflectivesurface (M3) is located upon the back layer of the display oriented toreflect light from the eyepiece into the exit pupil 202.

In some embodiments, the objective optics 220 receives tight of theexternal scene, and focuses the light of the external scene and forms asee-through image upon the spatial light modulator 240. The spatiallight modulator 240 modifies the see-through image to remove portions ofthe image that are to be occluded. The microdisplay 250 projects avirtual image to the beam splitter 230. The spatial light modulator 240transmits the modified see-through image to the beam splitter 230, wherethe beam splitter 230 merges the two images producing a combined imagein which the virtual image occludes portions of the see-through image.The beam splitter 230 then projects the combined image to the eyepiece210, whereupon the eyepiece projects the image to the exit pupil 202.

In some embodiments, the present invention comprises of an opticalsee-through head-mounted display 200, capable of combining an externalscene in the real world with a virtual view, where the opaqueness of theexternal scene is modulated and the digitally generated virtual viewoccludes parts of the external scene and vice versa. The inventioncomprises, a microdisplay 250 which transmits a virtual image, a spatiallight modulator 240 for modifying the light from an external scene, anobjective optics 220, which captures an external scene, a beamsplitter230 configured to merge the digitally generated virtual image from themicrodisplay 250 with the modified external scene from the spatial lightmodulator, an eyepiece 210 magnifying the virtual image and the modifiedexternal scene and an exit pupil 202 where the user observes a combinedview of the virtual image and the modified external scene.

In some embodiments, at least three reflective surfaces are used to foldthe virtual view path 205 and the see-through path 207 into two layers.The objective optics 220 is located on the front layer of the display,while the eyepiece 210 is located on the back layer of the display. Aseries of mirrors may be used to guide light along the optical pathsthrough the spatial light modulator, beam splitter and eyepiece. Thespatial light modulator 240 is located at or near an intermediate imageplane in the see-through path. The microdisplay 250 faces the beamsplitter 230, so that light from the microdisplay is transmitted intothe beam splitter 230. The beam splitter 230 combines light from themicrodisplay and the spatial light modulator and is oriented such thatthe direction of light transmission from the beam splitter is facing theeyepiece 210. The eyepiece 210 is located so that the light from thebeam splitter passed through the eyepiece and is transmitted into theexit pupil.

In some embodiments, the objective optics 220 receives an image of theexternal scene, and reflects or refracts the image to the spatial lightmodulator 240. The spatial light modulator 240 modifies the light fromthe external scene to remove portions of the image that are to beoccluded, and transmits or reflects the light into the beam splitter.The microdisplay 250 transmits a virtual image to the beam splitter 230,and the beam splitter 230 merges the two images producing a combinedimage in which the virtual image 205 occludes portions of the image ofthe external scene. The beam splitter 230 projects the combined image tothe eyepiece 210, which passes the image to the exit pupil 208. Thus theuser observes the combined image, in which the virtual image appears toocclude portions of the external scene.

FIG. 2 illustrates a schematic layout 200 in accordance with the presentinvention for achieving a compact OCOST-HMD system. In this exemplarylayout 200, the virtual view path. 205 (illustrated in dash lines)represents the light propagation path of the virtual view and comprisesof a microdisplay 250 for supplying display content and eyepiece 210through which a user views a magnified image of the displayed content;the see-through path 207 (illustrated in solid lines) represents thelight propagation path of the see-through view and comprises of bothobjective optics 220 and eyepiece 210 acting as a relay optics forpassing the light from an external scene in the real world to viewer'seye. To achieve a compact form factor and reduce the viewpoint offset,the see-through path 207 is folded into two layers in front of theviewer's eye through several reflective surfaces M1˜M3. The front layer215, accepting the incoming light from an external scene, containsmainly the objective optics 220 and necessary reflective surfaces M1 andM2. The back layer 217, coupling the light captured by the front layerinto a viewer's eye, mainly contains the eyepiece 210 and othernecessary optical components such as additional folding mirror M3, Inthe front layer 215, the reflective surface MI directs the incominglight from the external scene toward objective optics 220; and afterpassing through objective optics 220, the light is folded toward theback layer 217 through the reflective surface M2. The objective optics220 in the see-through path 207 forms at least one accessibleintermediate image. A spatial light modulator (SLM) 240 is placed at ornear the location of the accessible intermediate image, which istypically at the back focal plane of the objective optics, to provideopaqueness control and see-through modulation of the see-through view.In the present invention, a SIM is a light control device which canmodulates the intensity of the light beam that passes through or isreflected by it. A SLM can be either a reflection-type SLM, e.g., aliquid crystal on silicon (LCoS) display panel or a digital mirrordevice (DMD), or a transmission-type SLM, e.g., a liquid crystal display(LCD) panel. Both types of the SLM may be used for modulating thesee-through view for occlusion control in the see-through path 207. FIG.2(a) illustrates an exemplary configuration of using a reflection-typeSLM while FIG. 2(b) illustrates the use of a transmission-type SLM.Depending on the focal plane position of objective optics 220, the SLM240 can be placed at the position of SLM2 with a refection-type SLM inFIG. 2(a), or at the position of SLM1 with a transmission-type SLM inFIG. 2(b). The beamsplitter 230 folds the see-through path 207 andmerges it with the virtual view path 205 so that the same the eyepiece210 is shared for viewing the displayed virtual content and themodulated see-through view. The reflective surface M3 directs thevirtual view path 205 and see-through path 207 to exit pupil 202, wherethe viewer's eye observes a mixed virtual and real view. The reflectivesurfaces M1˜M3 could be either a standing alone element (e.g. mirror) orcould be strategically designed to be an integral part of the eyepiece210 or objective optics 220. The microdisplay 250 and SLIM 240 are bothlocated at the focal plane of the objective optics 220 and are opticallyconjugate to each other through the beamsplitter 230, which makes thepixel level opaqueness control on the see-through view possible, Thoughthe unit assembling the SLM 240, microdisplay 250, and beamsplitter 230is included in the back layer as shown in the exemplary figures, it maybe incorporated into the front layer when the back focal distance of theeyepiece is larger than that of the objective optics such that it ispreferred to place the combiner unit closer to the objective optics. Theapproach described above enables us to achieve a compact OCOST-HMDsolution and minimal view axis shift.

As one of its benefits, the optical layout 200 has applicability to manytypes of MOD optics, including, without limitation, rotationallysymmetric optics and non-rotationally symmetric freeform optics. Thereflective surfaces M1˜M3 for folding the optical paths may be planarmirrors, spherical, aspherical, or freeform surfaces with optical power.Some of the reflective surfaces may utilize freeform optical technology.Some of the reflective surfaces may also be strategically designed to bean integral part of the eyepiece 210 or objective optics 220 where thereflective surfaces not only facilitate the folding of the optical pathsfor achieving compact display design but also contribute optical powerand correct optical aberrations, In an exemplary configuration shown inFIG. 3, the present invention demonstrated the use of a one-reflectionfreeform prism as an eyepiece and objective optics where the prism is asingle optical element comprises of two refractive surfaces and onereflective surface for folding the optical path and correctingaberrations. In other examples of configurations, multi-reflectionfreeform prisms are demonstrated.

In another significant aspect of the present invention, besides theintermediate image accessible to the SLM 240, the see-through path 207may form additional intermediate images 260 by the objective optics 220,or eyepiece 210, or both. For instance, multiple-reflection freeformprism structure (typically more than 2) may be utilized as eyepiece orobjective optics, or both, which allow folding the see-through pathinside the objective and/or eyepiece prism multiple times and formintermediate image(s) inside the prism. As a result, the see-throughpath 207 may yield a total odd or even number of intermediate images.The potential advantage of creating more than one intermediate image isthe benefit of extended optical path length, long back focal distance,and the elimination of real-view erection element.

Depending on the total number of intermediate images being created andthe total number of reflective surfaces being used in the see-throughpath 207, a see-through view erection method may be needed to invertand/or revert the see-through view of the see-through path to maintainthe parity of the coordinate system of the see-through view and preventa viewer from seeing an inverted or reverted see-through view. As to thesee-through view erection method specifically, the present inventionconsiders two different image erection strategies. When a total evennumber of reflections is involved in the see-through path 207, whichinduces no change to the parity of the coordinate system of thesee-through view, the form of eyepiece 210 and objective optics 220 willbe designed such that an even number of intermediate images is createdin the see-through path 207. When an odd number of reflections existalong with an odd number of intermediate images in the see-through path207, which induces parity change, one of the reflective surfaces M1through M3 may be replaced by a roof mirror surface for the see-throughview erection. The preferred embodiments with the view erection using aroof reflection will be discussed below in connection with FIGS. 3 and5. The preferred embodiments with the view erection using anintermediate image will be discussed below in connection with FIGS. 4, 6and 7.

In one of its significant aspects, the present invention may utilizefreeform optical technology in eyepiece, objective optics or both toachieve a compact and lightweight OCOST-HMD. FIG. 3 shows a blockdiagram 300 of an exemplary approach to a compact OCOST-HMD design inaccordance with the present invention based on freeform opticaltechnology. The eyepiece 310 in the back layer 317 is a one-reflectionfreeform prism comprising three optical freeform surfaces: refractivesurface S1, reflective surface S2 and refractive surface 53. In virtualview path 305, the light ray emitted from microdisplay 350, enters theeyepiece 310 through the refractive surface S3, then is reflected by thereflective surface S2 and exits eyepiece 310 through the refractivesurface Si and reaches exit pupil 302, where the viewer's eye is alignedto see a magnified virtual image of microdisplay 350. The objectiveoptics 320 in the front layer 315 is also a one-reflection freeformprism comprising of three optical freeform surfaces: refractive surfaceS4, reflective surface S5 and refractive surface S6. In the see-throughpath 307, the objective optics 320 works together with eyepiece 310 actas a relay optics for the see-through view. The incoming light from anexternal scene reflected by mirror 325, enters the objective optics 320through the refractive surface S4, then is reflected by the reflectivesurface S5 and exits the objective optics 320 through refractive surfaceS6 and forms an intermediate image at its focal plane on SLM 340 forlight modulation. The beamsplitter 330 merges the modulated light in thesee-through path 307 with the light in the virtual view path 305 andfolds toward the eyepiece 310 for viewing. The beamsplitter 330 may be awire-grid type beamsplitter, a polarized cube beamsplitter or othersimilar type beamsplitters. In this approach, the SLM 340 is areflection-type SLM and is located at the SLM2 position of the schematiclayout 200 and is optically conjugated to the microdisplay 350 throughthe beamsplitter 330.

In this exemplary layout 300, the reflective surface M2 of the schematiclayout 200 is strategically designed to be an integrated part of theobjective prism 320 as freeform reflective surface S5; the reflectivesurface. M3 of the schematic layout 200 is strategically designed to bean integrated part of the eyepiece prism 310 as freeform reflectivesurface S2; the reflective surface M1 of schematic layout 200 isdesigned as a roof type mirror 325 for view erection given that thetotal number of reflections in see-through path 307 is 5 (an oddnumber).

In this exemplary layout 300, the eyepiece 310 and the objective optics320 may have an identical freeform prism structure. The advantage ofusing an identical structure for the eyepiece and the objective opticsis that the optical design strategy of one prism can be readily appliedto the other, which helps simplify the optical design. The symmetricstructure of the eyepiece and objective optics also helps correcting oddorder aberrations, such as coma, distortion, and lateral color.

FIG. 4 shows a block diagram 400 of another exemplary approach to acompact OCOST-HMD design in accordance with the present invention basedon freeform optical technology. In one exemplary implementation, theeyepiece 410 is a two-reflection prism and the objective optics 420 is afour-reflection prism. Inside the objective optics 420, an intermediateimage 460 is formed to erect the see-through view which eliminates thenecessity of using an erection roof reflective surface. The potentialadvantage of eliminating the erection prism is that this systemstructure may lead to a more compact design by folding the optical pathinside the objective prism multiple times. The eyepiece 410 in the backlayer 417 comprises of four optical freeform surfaces: refractivesurface Sit reflective surface 52, reflective surface S1′ and refractivesurface S3, In the virtual view path 405, the light ray emitted from themicrodisplay 450, enters eyepiece 410 through the refractive surface S3,then is consecutively reflected by the reflective surfaces S1′ and S2,and exits the eyepiece 410 through the refractive surface SI and reachesthe exit pupil 402, where the viewer's eye is aligned to see a magnifiedvirtual image of microdisplay 450. The refractive surface S1 and thereflective surface S1′ may be the same physical surfaces and possess thesame set of surface prescriptions. The objective optics 420 in the frontlayer 415 comprises of six optical freeform surfaces: refractive surfaceS4, reflective surfaces S5, S4′, 55′, and S6 and refractive surface S7.In the see-through path 407, the objective optics 420 works togetherwith the eyepiece 410 act as a relay optics for the see-through view.The incoming light from an external scene in the real world enters theobjective optics 420 through the refractive surface S4, then isconsecutively reflected by the reflective surfaces S5, S4′, S5′ and S6,and exits the objective optics 420 through the refractive surface S7 andforms an intermediate image at its focal plane on SLM 440 for lightmodulation. The refractive surface S4 and reflective surface S4′ may bethe same physical surfaces and possess the same set of surfaceprescriptions. The reflective surface S5 and the reflective surface S5′may be the same physical surfaces and possess the same set of surfaceprescriptions. The beamsplitter 430 merges the modulated light in thesee-through path 407 with the light in the virtual view path 405 andfolds toward the eyepiece 410 for viewing. The beamsplitter 430 may be awire-grid type beamsplitter, a polarized cube beamsplitter or othersimilar type beamsplitters. In this approach, the SLM 440 is areflection-type SLM and is located at the SLM2 position of the schematiclayout 200 and is optically conjugated to the microdisplay 450 throughbeamsplitter 430.

In this exemplary layout 400, the reflective surface M2 of the schematiclayout 200 is strategically designed as an integrated part of theobjective optics 420 as the reflective surface S6; the reflectivesurface M3 of the schematic layout 200 is strategically designed as anintegrated part of the eyepiece 410 as the reflective surface S2; thereflective surface M1 of schematic layout 200 is designed as anintegrated part of the objective optics 420 as the reflective surfaceS5. An intermediate image 460 is formed inside of the objective optics410 for the real-view erection. Given that the total number ofreflections in the see-through path 407 is 8 (an even number), no roofmirror is required on any reflective surfaces.

FIG. 5 shows a block diagram 500 of another exemplary approach to acompact OCOST-HMD design in accordance with the present invention basedon freeform optical technology. This approach facilitates the usage of atransmission-type SLM. The eyepiece 510 is a two-reflection prism andthe objective optics 520 is a one-reflection prism. A roof mirror 527 isplaced at the top of objective prism 520 to invert the see-through viewand to fold the see-through path 507 toward the back layer 517. Theeyepiece 510 in the back layer 517 comprises of four optical freeformsurfaces: refractive surface S1, reflective surface S2, reflectivesurface S1′ and refractive surface S3, In the virtual view path 505, thelight ray emitted from the microdisplay 550, enters the eyepiece 510through the refractive surface S3, then is consecutively reflected byreflective surfaces Sit and 52, and exits the eyepiece 510 through therefractive surface S1 and reaches exit pupil 502, where the viewer's eyeis aligned to see a magnified virtual image of the microdisplay 550. Therefractive surface Si and reflective surface S1′ may the same physicalsurfaces and possess the same set of surface prescriptions. Theobjective optics 520 in the front layer 515 comprises of three opticalfreeform surfaces: refractive surface S4, reflective surface S5 andrefractive surface 56. In the see-through path 507, the objective optics520 works together with the eyepiece 510 act as a relay optics for thesee-through view. The incoming light from an external scene in the realword enters the objective optics 520 through the refractive surface S4,then is reflected by the reflective surface S5 and exits the objectiveoptics 520 through the refractive surface S6 and is folded by the mirror527 toward the back layer 517 and forms an intermediate image at itsfocal plane on SLM 540 for light modulation. The beamsplitter 530 mergesthe modulated light in the see-through path 507 with the light in thevirtual view path 505 and folds the merged light toward the eyepiece 510for viewing. The beamsplitter 530 may be a wire-grid type beamsplitter,a polarized cube beamsplitter or other similar type beamsplitters. Inthis approach, the SLM 540 is a transmission-type SLM and is located atthe SLM, position of the schematic layout 200 and is opticallyconjugated to the micro-display 550 through the beamsplitter 530.

In this exemplary layout 500, the reflective surface M1 of the schematiclayout 200 is strategically designed as an integrated part of objectiveoptics 520 as the reflective surface S5; the reflective surface M3 ofthe schematic layout 200 is strategically designed as an integrated partof the eyepiece 510 as the reflective surface S2; the reflective surfaceM2 of the schematic layout 200 is designed as a roof type mirror 527 forview erection given that the total number of reflections in thesee-through path 507 is 5 (an odd number).

FIG. 6 shows a block diagram 600 of another exemplary approach to acompact OCOST-HMD design in accordance with the present invention basedon freeform optical technology. This approach also facilitates the usageof a transmission type SLM. In one exemplary implementation, theeyepiece 610 is a two-reflection freeform prism and the objective optics620 is a three-reflection freeform prism. Inside the objective optics620, an intermediate image 660 is formed to erect the see-through view.The eyepiece 610 in the back layer 617 comprises of four opticalfreeform surfaces: refractive surface S1, reflective surface S2,reflective surface S1′ and refractive surface S3. In the virtual viewpath 605, the light ray emitted from the microdisplay 650, enters theeyepiece 610 through the refractive surface S3, then is consecutivelyreflected by reflective surfaces S1′ and S2, and exits the eyepiece 610through the refractive surface S1 and reaches exit pupil 602, where theviewer's eye is aligned to see a magnified virtual image of themicrodisplay 650. The refractive surface S1 and the reflective surfaceS1′ may the same physical surfaces and possess the same set of surfaceprescriptions. The objective optics 620 in the front layer 615 comprisesof five optical freeform surfaces: refractive surface S4, reflectivesurfaces S5, S4′ and S6 and refractive surface S7, In the see-throughpath 607, the objective optics 620 works together with the eyepiece 610acting as relay optics for the see-through view. The incoming light froman external scene in the real world enters the objective optics 620through the refractive surface S4, consecutively reflected by thereflective surfaces S5, S4′ and S6, and exits the objective optics 620through the refractive surface S7 and forms an intermediate image at itsfocal plane on SLM 640 for light modulation. The refractive surface S4and the reflective surface S4′ may be the same physical surfaces andpossess the same set of surface prescriptions. The beamsplitter 630merges the modulated light in the see-through path 607 with the light inthe virtual view path 605 and folds toward the eyepiece 610 for viewing.The beamsplitter 630 may be a wire-grid type beamsplitter, a polarizedcube beamsplitter or other similar type beamsplitters in this approach,the SLM 640 is a transmission-type SLM and is located at the SLM1position of the schematic layout 200 and is optically conjugated to themicro-display 650 through the beamsplitter 630.

In this exemplary layout 600, the reflective surface M1 of the schematiclayout 200 is strategically designed as an integrated part of theobjective optics 620 as the reflective surface S5; the reflectivesurface M2 of the schematic layout 200 is strategically designed as anintegrated part of the objective optics 620 as the reflective surfaceS6; the reflective surface M3 of the schematic layout 200 isstrategically designed as an integrated part of the eyepiece 610 as thereflective surface S2. An intermediate image 660 is formed inside of theobjective optics 610 for real-view erection. Given that the total numberof reflections in the see-through path 607 is 6 (an even number), noroof mirror is required on any reflective surface,

FIG. 7 shows a block diagram 700 of another exemplary approach to acompact OCOST-HMD design in accordance with the present invention basedon freeform optical technology. In one exemplary implementation, boththe eyepiece and the objective optics are two-reflection freeform prismsand have nearly identical structure. The advantage of using an identicalstructure for the eyepiece and objective is that the optical designstrategy of one prism can be readily applied to the other, which helpssimplify the optical design. The symmetric structure of the eyepiece andobjective prisms may also help correcting odd order aberrations, such ascoma, distortion, and lateral color. The eyepiece 710 in the back layer717 comprises of four optical freeform surfaces: refractive surface S1,reflective surface S2, reflective surface S1′ and refractive surface S3.In the virtual view path 705, the light ray emitted from themicrodisplay 750, enters the eyepiece 710 through the refractive surfaceS3, then is consecutively reflected by the reflective surfaces S1′ andS2, and exits the eyepiece 710 through the refractive surface S1 andreaches exit pupil 702, where the viewer's eye is aligned to see amagnified virtual image of the microdisplay 750. The refractive surfaceS1 and the reflective surface S1′ may the same physical surfaces andpossess the same set of surface prescriptions, The objective optics 720in the front layer 715 comprises of four optical freeform surfaces:refractive surface S4, reflective surfaces S5, S4′ and refractivesurface S6. In the see-through path 707, the objective optics 720 workstogether with the eyepiece 710 acting as a relay optics for thesee-through view. The incoming light from an external scene in the realworld enters the Objective optics 720 through the refractive surface S4,consecutively reflected by the reflective surfaces S5, S4′, and exitsthe objective optics 720 through the refractive surface S6 and forms anintermediate image 760 at its focal plane. The beamsplitter 780 foldsthe see-through path 707 away from the back layer 715 toward the mirror790 positioned at the focal plane of the objective optics 720. Thesee-through path 707 is reflected by the mirror 790 back toward the backlayer 715. A relay lens 770 is used to create an image of theintermediate image 760 at the SLM2 position of the schematic layout 200for view modulation. The beamsplitter 730 merges the modulated light inthe see-through path 707 with the light in the virtual view path 705 andfolds toward the eyepiece 710 for viewing. In this approach, the SLM 740is a reflection-type SLM and is optically conjugated to the microdisplay750 through beamsplitter 730. Due to the fact that the intermediateimage 760 is optically conjugated to the SLM 740, the positions of theSIM 740 and the mirror 790 are interchangeable.

In this exemplary layout 700, the reflective surface M1 of the schematiclayout 200 is strategically designed as an integrated part of theobjective optics 720 as the reflective surface S5; the reflectivesurface M3 of the schematic layout 200 is strategically designed as anintegrated part of the eyepiece 710 as the reflective surface S2; thereflective surface M2 of the schematic layout 200 is positioned at thefocal plane of the Objective optics 710 as the mirror 790 and folds thesee-through path 707 toward the virtual view path 705; The intermediateimage 760 is formed at the focal plane of the objective optics 720 forreal-view erection. Given that the total number of reflections in thesee-through path 707 is 8 (an even number), no roof mirror is requiredon any reflective surface.

FIG. 8 schematically illustrated an exemplary design 800 based on theexemplary approach depicted in FIG. 3. The design achieved a diagonalFOV of 40 degrees, which is 31.7 degrees in the horizontal direction(X-axis direction) and 25.6 degrees in the vertical direction (Y-axisdirection), an exit pupil diameter (ETD) of 8 mm (non-vignetted), and aneye clearance of 18 mm. The design is based on a 0.8″ microdisplay witha 5:4 aspect ratio and a 1280×10²⁴ pixel resolution. The microdisplayhas an effective area of 15.36 mm and 12.29 mm and a pixel size of 12 m.The design used a SLM of the same size and resolution as themicrodisplay. A polarized cube beamsplitter is used to combine thevirtual view path and the see-through path. DOE plates 882 and 884 areused to correct chromatic aberrations. The system is measured as 43mm(X)×23 mm (Y)×44.5 mm (Z). The viewpoint shifts between the entrancepupil 886 and exit pupil 802 are 0.6 mm in Y direction and 67 mm in Zdirection, respectively.

An exemplary optical prescription of the eyepiece 810 is listed in theTables 1-4. All the three optical surfaces in the eyepiece 810 areanamorphic aspheric surface (AAS), The sag of an AAS surface is definedby

${z = {\frac{{c_{x}x^{2}} + {c_{y}y^{2}}}{1 + \sqrt{1 - {( {1 + K_{x}} )c_{x}^{2}y^{2}}}} + {{AR}\{ {{( {1 - {AP}} )x^{2}} + {( {1 + {AP}} )y^{2}}} \}^{2}} + \{ {{( {1 - {BP}} )x^{2}} + {( {1 + {BP}} )y^{2}}} \}^{3} + {{CR}\{ {{( {1 - {CP}} )x^{2}} + {( {1 + {CP}} )y^{2}}} \}^{4}} + {{DR}\{ {{( {1 - {DP}} )x^{2}} + {( {1 + {DP}} )y^{2}}} \}^{5}}}},$where z is the sag of the free-form surface measured along the z-axis ofa local x, y, z coordinate system, c_(x) and c_(y) are the vertexcurvature in x and y axes, respectively, K_(x) and K_(y) are the conicconstant in x and y axes, respectively, AR, BR, CR and DR are therotationally symmetric portion of the 4th, 6th, 8th, and 10th orderdeformation from the conic, AP, BP, CP, and DP are the non-rotationallysymmetric components of the 4th, 6th, 8th, and 10th order deformationfrom the conic.Table 1: Optical surface prescription of surface 1 of the eyepieceprism, See FIG. 13.Table 2: Optical surface prescription of surface 2 of the eyepieceprism, See FIG. 14Table 3: Optical surface prescription of surface 3 of the eyepieceprism, See FIG. 15Table 4: Position and orientation parameters of the eyepiece prism, SeeFIG. 16

An exemplary optical prescription of the objective optics 820 is listedin the Tables 5-8. All the three optical surfaces in the objectiveoptics 820 are anamorphic aspheric surface (AAS).

Table 5: Optical surface prescription of surface 4 of the objectiveprism, See FIG. 17.

Table 6: Optical surface prescription of surface 5 of the objectiveprism, See FIG. 18.

Table 7: Optical surface prescription of surface 6 of the objectiveprism, See FIG. 19.

Table 8: Position and orientation parameters of the objective prism, SeeFIG. 20.

An exemplary optical prescription of the DOE plate 882 and 884 is listedin the Tables 9.

Table 9: Surface parameters for DOE plates 882 and 884. See FIG. 21.

FIG. 9 shows the field map of polychromatic modulation transferfunctions (MTF) of the virtual display path at cutoff frequency 401ps/min (line pairs per millimeter) evaluated using 3 mm pupil diameter.The 401 ps/mm cutoff frequency was determined from the pixel size of themicrodisplay. The plot shows that our design has very good performancefor majority fields except two upper display corners whose MTF values atcutoff frequency are little less than 15%). Across the entire FOV thedistortion of the virtual display path is less than 2.9%, while thedistortion of the see-through path is less than 0.5%. The totalestimated weight for the optics alone is 33 grams per eye.

FIG. 10 schematically illustrated an exemplary design 1000 based on theexemplary approach depicted in FIG. 3. The design achieved a diagonalFOV of 40 degrees with 35.2 degrees horizontally (X-direction) and 20.2degrees vertically (V-direction), an exit pupil diameter (EPD) of 8 ram(non-vignetted), and an eye clearance of 18 mm. The design is based on a0.7″ microdisplay with a 16:9 aspect ratio and a 1280×720 pixelresolution. The design used a SLM of the same size and resolution as themicrodisplay. A wire-grid plate beamsplitter is used to combine thevirtual view path and the see-through path. The same freeform prism isused as the eyepiece and the objective optics.

An exemplary optical prescription of the freeform prism is listed in theTables 10-15. Two surfaces in the prism are anamorphic aspheric surface(AAS) and one is aspheric surface (ASP). The sag of an ASP surface isdefined by

$z = {\frac{{cr}^{2}}{1 + \sqrt{1 - {( {1 + K} )c^{2}r^{2}}}} + {Ar}^{4} + {Br}^{6} + {Cr}^{8} + {Dr}^{10} + {Er}^{12} + {{Fr}^{14}{Gr}^{16}} + {{Hr}^{18}{Jr}^{20}}}$where z is the sag of the surface measured along the z-axis of a localx, y, z coordinate system, c is the vertex curvature, k is the conicconstant, A through J are the 4th, 6th, 8th, 10th, 12th, 14th, 16th,18th, and 20th order deformation coefficients, respectively.Table 10: Optical surface prescription of surface 1 of the freeformprism, See FIG. 22Table 11: Optical surface prescription of surface 2 of the freeformprism, See FIG. 23.Table 12: Optical surface prescription of surface 3 of the freeformprism, See FIG. 24.Table 13: Position and orientation parameters of the freeform prism asthe eyepiece, See FIG. 25.

FIG. 11 shows the field map of polychromatic modulation transferfunctions (MTF) of the virtual display path at cutoff frequency 401psi/mm (line pairs per millimeter) evaluated using 3 mm pupil diameter.The plot shows that our design has very good performance for majorityfields.

FIG. 12 depicts a block diagram of an example of an image processingpipeline necessary for the present invention. Firstly, the depth map ofthe external scene is extracted using proper depth sensing means. Then,the virtual object is compared with the depth map to determine theregions where the occlusion occurs. A mask generation algorithm createsa binary mask image according to the pre-determined occlusion regions.The mask image is then displayed on spatial light modulator to block thelight from the occluded region in the intermediate image of the externalscene. A virtual image of the virtual object is rendered and displayedon the micro-display. The viewer observes a combined image of thevirtual image and the modulated see-through image of the external scenethrough the display device of the present invention,

Compared to the prior art, the present invention features a folded imagepath that permits the invention to be compressed into a compact form,more easily wearable as a head-mounted display. In the prior art (U.S.Pat. No. 7,639,208 B1), the optical path is linearly arranged usingrotationally symmetric lenses. As a result the prior art occlusion-typedisplays have a long telescope-like shape, which is unwieldy for wearingon the head. The present invention folds the image path using reflectivesurfaces into two layers to that the spatial light modulator,microdisplay and beamsplitter, are mounted to the top of the head,rather than linearly in front of the eye.

The prior art relies on only a reflection type spatial light modulator,while the present invention may use either a reflection or transmissiontype spatial light modulator. Moreover, the prior art requires apolarized beamsplitter to modulate the external image, while the presentinvention does not necessitate polarization.

Since the present invention is arrange in layers, the eyepiece and theobjective optics are not necessarily collinear, as in the case in theprior art. The objective optics is also not necessarily tele-centric.

In the prior art, due to the optics of the system the view of the worldis a mirror reflection of the see-through view. The present inventionthe folded image path allows a roof mirror to be inserted to maintainparity between the view of the user and the external scene. This makesthe present invention more functional from the user's perspective.

Compared to the prior art, the present invention makes use of freeformoptical technology, which allows the system to be made even morecompact. The freeform optical surfaces can be designed to reflect lightinternally multiple times, so that mirrors may not be needed to fold thelight path.

In the present invention, the reflective surfaces for folding theoptical paths may be planar mirrors, spherical, aspherical, or freeformsurfaces with optical power. A significant aspect of the presentinvention lies in that some of the reflective surfaces utilize freeformoptical technology, which helps to boost the optical performance andcompactness. In the present invention, sonic of the reflective surfacesare strategically designed to be an integral part of the eyepiece orobjective optics where the reflective surfaces not only facilitate thefolding of the optical path for achieving compact display design butalso contribute optical power and correct optical aberrations. Forexample, in FIG. 2, the reflective surfaces M1˜M3 were shown as genericmirrors separate from the eyepiece and objective optics. In FIG. 3, twoof the mirrors (M2 and M3) are freeform surfaces incorporated into thefreeform eyepiece and Objective prisms as S2 and S5. In FIG. 4, 4reflective freeform surfaces were incorporated into the freeformobjective prism and 2 were incorporated into the freeform eyepieceprisms. In FIG. 5, 1 freeform surface was in the objective prism, 2freeform surfaces were in the eyepiece, in addition to a roof prism. InFIG. 6, 3 freeform surfaces are in the objective and 2 freeform surfacesin the eyepiece. In FIG. 7, 2 reflective freeform mirrors are in theobjective, 2 freeform mirrors are in the eyepiece, in addition to amirror 790 and a beamsplitter 780.

Our invention ensures that the see-through view seen through the systemis correctly erected (neither inverted nor reverted). Two differentoptical methods were utilized in our embodiments for achieving this,depending on the number of intermediate images formed in the see-throughpath and the number of reflections involved in the see-through path. Inthe case of an odd number of intermediate images, an optical method isprovided to invert and/or revert the see-through view in the see-throughpath. For example, depending on the number of reflections involved inthe see-through path, examples of the possible methods include, but notlimited to, inserting an additional reflection or reflections, utilizinga roof mirror surface, or inserting an erector lens, in the case of aneven number of intermediate images, no image erection element is neededif no parity change is needed. For instance, multiple-reflectionfreeform prism structure (typical more than 2) may be utilized aseyepiece or objective optics, or both, which allow folding thesee-through optical path inside the objective and/or eyepiece prismmultiple times and form intermediate image(s inside the prism to erectthe see-through view which eliminates the necessity of using an erectionroof reflective surface.

In FIG. 3, only 1 intermediate image is created in the see-through path.This structure utilized a roof prism for 325 to properly create anerected see-through view.

In FIG. 4, a 4-reflection freeform prism was utilized as an objectiveoptics, which created 2 intermediate images (one for SLM 440, and one460 inside the prism). Additionally, there were total 8 reflectionsinvolved in the see-through path, which leads to no parity change.Therefore, an erected view is created, It is worth mention that thestructure of the objective and eyepiece may be exchanged for the sameresults.

In FIG. 5, 1 intermediate image is created in the see-through path forthe SIM. This design utilized a roof prism 527 to erect the see-throughview.

In FIG. 6, a 3-reflection freeform prism was utilized as an objectiveoptics, which created 2 intermediate images (one for SLM 640, and one660 inside the prism). Additionally, there were total 6 reflectionsinvolved in the see-through path, which leads to no parity change.Therefore, an erected view is created, It is worth mention that thestructure of the objective and eyepiece may be exchanged for the sameresults.

In FIG. 7, the objective optics 720 utilized only 2 reflections, thecombination of the beamsplitter 780 and the mirror 790 facilitated thecreation of 2 intermediate images in the see-through path (one for theSLM 740 and an additional one 760). Additionally, total 8 reflectionswere involved in the see-through path. Therefore, en erected see-throughview was created.

It is very important for a see-through head mounted display to maintainthe parity of the external scene which provides the users a realisticexperience as their usual views without a HMD.

Although there has been shown and described the preferred embodiment ofthe present invention, it will be readily apparent to those skilled inthe art that modifications may be made thereto which do not exceed thescope of the appended claims. Reference numbers recited in the claimsare exemplary and for ease of review by the patent office only, and arenot limiting in any way. In some embodiments, the figures presented inthis patent application are drawn to scale, including the angles, ratiosof dimensions, etc. In some embodiments, the figures are representativeonly and the claims are not limited by the dimensions of the figures.

The reference numbers recited in the below claims are solely for ease ofexamination of this patent application, and are exemplary, and are notintended in any way to limit the scope of the claims to the particularfeatures having the corresponding reference numbers in the drawings.

What is claimed:
 1. A compact optical see-through head-mounted display,capable of combining a see-through path with a virtual view path suchthat the opaqueness of a see-through view can be modulated and a virtualview occludes parts of the see-through view and vice versa, the displaycomprising: a. a microdisplay for generating an image to be viewed by auser, the microdisplay having a virtual view path associated therewith;b. a spatial light modulator for modifying light from an external scenein the real world to block portions of the see-through view that are tobe occluded, the spatial light modulator having a see-through pathassociated therewith; c. an objective optics configured to receive theincoming light from the external scene and to focus the light upon thespatial light modulator; d. a beamsplitter configured to merge a virtualimage from a microdisplay and a modulated see-through image of anexternal scene passing from a spatial light modulator, producing acombined image; e. an eyepiece configured to magnify the combined image;f. an exit pupil configured to face the eyepiece, the exit pupilwhereupon the user observes a combined view of the virtual andsee-through views in which the virtual view occludes portions of thesee-through view; and g. a plurality of reactive surfaces configured tofold the virtual view path and the see-through path into two layers,wherein a first reflective surface is disposed upon a front layer of thedisplay oriented to reflect the light from the external scene, whereinthe objective optics is disposed upon the front layer of the display,wherein a second reflective surface is disposed upon the front layer ofthe display oriented to reflect the light into the spatial lightmodulator, where the spatial light modulator is disposed at or near anintermediate image plane of the see-through path, in opticalcommunication with the objective optics and the eyepiece through thebeamsplitter along the see-through path, where the microdisplay isdisposed at a focal plane of the eyepiece along the virtual view path,in optical communication with the eyepiece through the beamsplitter,where the, beam splitter is disposed such that the see-through path ismerged with the virtual view path and the light from both thesee-through path and the virtual view path is directed to the eyepiece,wherein the eyepiece is disposed upon a back layer of the display,wherein a third reflective surface is disposed upon the back layer ofthe display oriented to reflect the light from the eyepiece into theexit pupil; whereupon the objective optics receives light of theexternal scene, whereupon the objective optics focus the light of theexternal scene and forms a see-through image upon the spatial lightmodulator, whereupon the spatial light modulator modifies thesee-through image to remove portions of the image that are to beoccluded, whereupon the microdisplay projects a virtual image to thebeam splitter, whereupon the spatial light modulator transmits themodulated see-through image to the beam splitter, whereupon the beamsplitter merges the virtual and see-through images producing a combinedimage in which the virtual image occludes portions of the see-throughimage, whereupon the beam splitter projects the combined image to theeyepiece, whereupon the eyepiece projects the combined image to the exitpupil, whereupon the user observes the combined image, in which thevirtual image occludes portions of the external scene.
 2. The display ofclaim 1, where the spatial light modulator is a transmission typespatial light modulator, where the spatial light modulator is disposedin front of the beam splitter, whereupon the light from the objectiveoptics passes through the spatial light modulator before reaching thebeam splitter, wherein the opaqueness of the spatial light modulator iscontrolled to block light from portions of the external scene.
 3. Thedisplay of claim 1, where the spatial light modulator is a reflectiontype spatial light modulator, where the spatial light modulator isdisposed behind the beam splitter, whereupon light from the objectiveoptics passes through the beam splitter and is reflected back from thespatial light modulator to the beam splitter, wherein a reflectivity ofthe spatial light modulator is controlled to reflect only light fromparts of the external scene not to be occluded.
 4. The display of claim1, where an intermediate image is formed at one or more points in thesee-through path, where the spatial light modulator is disposed at ornear one of the intermediate image planes.
 5. The display of claim 1,where one or more of the first, second, and third reflective surfacesare stand-alone surfaces with optical power to fold the see-through andvirtual view paths and focus the light.
 6. The display of claim 1, whereone or more of the first, second, and third reflective surfaces arefreeform surfaces.
 7. The display of claim 1, where the first and/orsecond reflective surfaces in the front layer are contained within theobjective optics.
 8. The display of claim 1, where the third reflectivesurface in the back layer is contained within the eyepiece.
 9. Thedisplay of claim 1, where the objective optics is a freeform prismformed by a plurality of reflective and refractive surfaces to image theexternal scene into the spatial light modulator.
 10. The display ofclaim 1, where the eyepiece is a freeform prism formed by a plurality ofreflective and refractive surfaces to magnify the virtual image and themodulated see-through image.
 11. The display of claim 9, where the firstand/or the second reflective surface in the front layer is containedwithin the objective optics.
 12. The display of claim 10, where thethird reflective surface in the back layer is contained within theeyepiece.
 13. The display of claim 1, where an even number ofintermediate images are formed along the see-through path to invert thesee-through view so as to maintain parity between the external scene andthe see-through view presented to a viewer.
 14. The display of claim 1,where one of the first, second, and third reflective surfaces isreplaced by a roof mirror in order to revert the see-through view so asto maintain parity between the external scene and the see-through viewpresented to a viewer.
 15. The display of claim 1, where both theeyepiece and the objective optics have identical optical structure. 16.The display of claim 15, where both the eyepiece and the objectiveoptics are freeform prisms of identical shape.
 17. The display of claim1 where the beamsplitter is disposed upon the front layer.
 18. Thedisplay of claim 1, where one or more diffractive optical element platesare placed in an optical path to correct chromatic aberrations.
 19. Thedisplay of claim 1, wherein the objective optics is a one-reflectionprism comprising three optical surfaces: a first refractive surface, athird reflective surface and a second refractive surface, and whereinthe eyepiece is a one-reflection prism comprising three opticalsurfaces: a third refractive surface, a fourth reflective surface and afourth refractive surface, where the second reflective surface iscontained within the objective optics and the third reflective surfaceis contained within the eyepiece, where a roof mirror replaces the firstreflective surface to invert the see-through view, and a reflection-typespatial light modulator is used to modulate the light from the externalscene.
 20. The display of claim 19, whereupon the incoming light fromexternal scene reflected by the roof mirror, enters the objective opticsthrough the first refractive surface, then is reflected by the thirdreflective surface and exits the objective optics through the secondrefractive surface and forms an intermediate image at its focal plane onthe spatial light modulator, whereupon the spatial light modulatormodulates the light in the see-through path to block the light to beoccluded, whereupon the spatial light modulator reflects the modulatedlight into the beam splitter, whereupon the light from the microdisplayenters the beam splitter, whereupon the beamsplitter merges themodulated light in the see-through path with the light in the virtualview path and folds toward the eyepiece for viewing, whereupon the lightfrom the beamsplitter enters the eyepiece through the fourth refractivesurface, that is reflected by the fourth reflective surface and exitsthe eyepiece through the third refractive surface and reaches the exitpupil, where a viewer's eye is aligned to see a combined view of avirtual view and a modulated see-through view.
 21. The display of claim1, where the objective optics is a four-reflection freeform prismcomprising six optical surfaces: refractive surface, reflective surfacesand refractive surface, where the eyepiece is a two-reflection prismcomprising four optical surfaces: refractive surface, reflectivesurface, reflective surface and refractive surface, where the firstreflective surface and the second reflective surfaces are containedwithin the objective optics and the third reflective surface iscontained within the eyepiece, where the objective optics forms anintermediate image inside the objective optics, where an even number ofintermediate images are formed along the see-through path to invert thesee-through view, where a reflection-type spatial light modulator isused to modulate the light from the external scene.
 22. The display ofclaim 21, whereupon the incoming light from an external scene entersobjective optics through the refractive surface, then is consecutivelyreflected by the reflective surfaces, and exits the objective opticsthrough the refractive surface, whereupon the incoming light forms anintermediate image at its focal plane on the spatial light modulator,whereupon the spatial light modulator modulates the light in thesee-through path to block the light to be occluded, whereupon thespatial light modulator reflects the modulated light into the beamsplitter, whereupon the light from the microdisplay enters the beamsplitter, whereupon the beamsplitter merges the modulated light in thesee-through path with the light in the virtual view path and foldstoward the eyepiece for viewing, whereupon the light from the beamsplitter enters the eyepiece through the refractive surface, then isconsecutively reflected by the reflective surfaces and exits theeyepiece through the refractive surface Si and reaches the exit pupil,where the viewer's eye is aligned to see a combined view of a virtualview and a modulated see-through view.
 23. The display of claim 21,where the refractive surface and reflective surface are the samephysical surfaces and possess the same set of surface prescriptions. 24.The display of claim 1, where the objective optics is a one-reflectionprism comprising three optical surfaces: refractive surface, reflectivesurface and refractive surface, where the eyepiece is a two-reflectionprism comprising four optical surfaces: refractive surface, reflectivesurface, reflective surface and refractive surface, where the firstreflective surface is contained within the objective optics and thethird reflective surface is contained within the eyepiece, where a roofmirror replaces the second reflective surface to invert the see-throughview, where a transmission-type spatial light modulator is used tomodulate the light from the external scene.
 25. The display of claim 24,whereupon incoming light from an external scene enters the objectiveoptics through the refractive surface, then is reflected by thereflective surface and exits the objective optics through the refractivesurface and is folded by the mirror toward the back layer and forms anintermediate image at its focal plane on the spatial light modulator,whereupon the spatial light modulator modulates the light in thesee-through path to block the light to be occluded, whereupon thespatial light modulator transmits the modulated light into the beamsplitter, whereupon the light from the microdisplay enters the beamsplitter, whereupon the beamsplitter merges the modulated light in thesee-through path with the light in the virtual view path and foldstoward the eyepiece for viewing, whereupon the light from the beamsplitter enters the eyepiece through the refractive surface, then isconsecutively reflected by reflective surfaces, and exits the eyepiecethrough the refractive surface and reaches exit pupil, where theviewer's eye is aligned to see a combined view of a virtual view and amodulated see-through view.
 26. The system of claim 24, where surfaceand reflective surface are the same physical surfaces and possess thesame set of surface prescriptions.
 27. The display of claim 1, where theobjective optics is a three-reflection prism comprising five opticalfreeform surfaces: refractive surface, reflective surface and refractivesurface, where the eyepiece is a two-reflection prism comprising fouroptical surfaces: refractive surface, reflective surface, reflectivesurface and refractive surface, where the first reflective surface andthe second reflective surface are contained within the objective opticsand the third reflective surface is contained within the eyepiece, wherethe objective optics forms an intermediate image inside the objectiveoptics, where an even number of intermediate images are formed along thesee-through path to invert the see-through view, where atransmission-type spatial light modulator is used to modulate the lightfrom the external scene.
 28. The display of claim 27, whereupon theincoming light from an external scene enters objective optics throughthe refractive surface, is consecutively reflected by reflectivesurfaces, and exits the objective optics through the refractive surfacewhereupon the incoming light forms an intermediate image at its focalplane on the spatial light modulator, whereupon the spatial lightmodulator modulates the light in the see-through path to block the lightto be occluded, whereupon the spatial light modulator transmits themodulated light into the beam splitter, whereupon the light from themicrodisplay enters the beam splitter, whereupon the beamsplitter mergesthe modulated light in the see-through path with the light in thevirtual view path and folds toward the eyepiece for viewing, whereuponthe light from the beam splitter enters the eyepiece through therefractive surface, then is consecutively reflected by the reflectivesurfaces, and exits the eyepiece through the refractive surface Si andreaches the exit pupil, where the viewer's eye is aligned to see acombined view of a virtual view and a modulated see-through view. 29.There system of claim 27, where the refractive surface and thereflective surface may the same physical surfaces and possess the sameset of surface prescriptions.
 30. There system of claim 27, where therefractive surface and the reflective surface are the same physicalsurfaces and possess the same set of surface prescriptions.
 31. Thedisplay of claim 1, where the objective optics is a two-reflection prismcomprising our optical freeform surfaces: refractive surface, reflectivesurface, reflective surface and refractive surface, where the eyepieceis a two-reflection prism comprising four optical surfaces: refractivesurface, reflective surface, reflective surface and refractive surface,where the first reflective surface is contained within the objectiveoptics and the third reflective surface is contained within theeyepiece, where the second reflective surface as the minor along with afirst beamsplitter and a relay lens is configured to fold thesee-through path and create a secondary intermediate image, where aneven number of intermediate images are formed along the see-through pathto invert the see-through view, where a reflection-type spatial lightmodulator is used to modulate the tight from the external scene.
 32. Thedisplay of claim 31, whereupon the incoming light from an external sceneenters the objective optics through refractive surface, consecutivelyreflected by the reflective surfaces, and exits the objective opticsthrough the refractive surface, whereupon the incoming light isreflected by the first beamsplitter onto the mirror, where it forms anintermediate image, whereupon the mirror reflects the light from thefront layer onto the relay lens, whereupon the relay lens forms anotherintermediate image upon the spatial light modulator whereupon thespatial tight modulator modulates the light in the see-through path toblock the light to be occluded, whereupon the spatial light modulatorreflects the modulated light into the second beam splitter, whereuponthe light from the microdisplay enters the second beam splitter,whereupon the second beam splitter merges the modulated light in thesee-through path with the light in the virtual view path and foldstoward the eyepiece for viewing, whereupon the light from the beamsplitter enters the eyepiece through refractive surface, then isconsecutively reflected by the reflective surfaces, and exits theeyepiece through the refractive surface Si and reaches exit pupil, wherethe viewer's eye is aligned to see a combined view of a virtual view anda modulated see-through view.
 33. The display of claim 31, where thepositions of the mirror and the spatial light modulator areinterchangeable.
 34. The display of claim 19, where one or more of thesee-through and virtual view surfaces of the objective optics is anaspherical surface with or without rotational symmetry.
 35. The displayof claim 19, where one or more of the see-through and virtual viewsurfaces of the eyepiece prism is an aspherical surface with or withoutrotational symmetry.
 36. The display of claim 1, where the beamsplitteris in form of a cube or a plate and could be a non-polarizedbeamsplitter or a polarized beamsplitter.
 37. The display according toclaim 1, where said compact optical see-through head mounted display isone of a monocular and a binocular.